Artificial intelligence systems and methods for interior design

ABSTRACT

Systems and methods for generating a furnishing plan for a property are disclosed. An exemplary system includes a communication interface configured to receive a floor plan of the property and a neural network model. The system further includes at least one processor configured to obtain structural data of the property based on the floor plan and learn furnishing information by applying the neural network model to the floor plan and the structural data. The furnishing information identifies one or more furnishing objects, positions of the respective furnishing objects placed in the floor plan, and dimensions of the respective furnishing objects. The at least one processor is also configured to generate the furnishing plan for the property based on the furnishing information.

RELATED APPLICATIONS

This application hereby claims the benefits of priority to ChineseApplication No. 201910636694.0 filed on Jul. 15, 2019, ChineseApplication No. 201910637657.1 filed on Jul. 15, 2019, ChineseApplication No. 201910637579.5 filed on Jul. 15, 2019, and ChineseApplication No. 201910637659.0 filed on Jul. 15, 2019, all of which arehereby incorporated by reference in their entireties.

TECHNICAL FIELD

The present disclosure relates to systems and methods for interiordesign, and more particularly, to artificial intelligence systems andmethods for generating interior design plans (e.g., remodeling and/orfurnishing plans) and visualization of such interior design.

BACKGROUND

Property owners may need assistance with interior design at variousoccasions, such as, when they would like to remodel a space, refurnish aspace, or stage a space before putting the property on the market. Onechallenge with interior design has been imposed by the difficulty toaccess how the design plan may adapt to the actual space before the planis executed. For example, when a person is browsing online to search fora piece of furniture for his living room, while the furniture may bewell depicted with multiple images or videos, the user could notvisualize how it may fit into his living room. It is usually not clearto the property owner until the piece of furniture is purchased andplaced into the living room that the dimensions of the piece may not fitor the style of the piece does not match with other decorations in theroom.

In addition to furnishing, interior remodeling also requires theremodeling plan to adapt to the actual floor plan. For example, theremodeling design should take into consideration such as the size of thespace, the layout, intended function of the space, and furnishingpreferences, etc. Sometimes, during a kitchen remodeling project, it ishard for the property owner to decide whether to knock off a wall toreduce it to a half wall, let alone how to make it happen.

Therefore, interior designing can greatly benefit from intelligentlygenerated design plans (e.g., remodeling plans or furnishing plans) andthe ability to visualize the design in the actual space before thedesign will be implemented in that space. To address these needs,embodiments of the disclosure provide artificial intelligence systemsand methods for generating interior design plans (e.g., remodelingand/or furnishing plans) and visualization of such interior design.

SUMMARY

In one aspect, embodiments of artificial intelligence systems, methods,computer-readable medium for visualizing furnishing objects in an imageof an interior space are disclosed. In some embodiments, the image ofthe interior space may be captured by a 3D scanner and include existingfurnishing objects. The existing furnishing objects may be removed andthe image may be restored by filling the holes left after removing thefurnishing objects. One or more new furnishing objects may be insertedto the restored image and the placement of the furnishing objects in theimage may be adjusted, before the new image is provided to a user.

In another aspect, embodiments of artificial intelligence systems,methods, computer-readable medium for suggesting new furnishing objectsfor an interior space are disclosed. In some embodiments, the image ofthe interior space may be captured by a 3D scanner and include existingfurnishing objects. Feature information of the existing furnishingobjects in the image may be determined using a learning model. Dimensioninformation of the existing furnishing objects may be determined basedon 3D point cloud data. Target furnishing objects that do not match withthe interior space may be identified based on attributes of thefurnishing objects determined based on the feature information and/orthe dimension information. New furnishing objects may be selected andsuggested to a user to replace the nonmatched furnishing objects.

In yet another aspect, embodiments of artificial intelligence systems,methods, computer-readable medium for generating a remodeling plan for aproperty are disclosed. In some embodiments, structural data may beobtained based on a floor plan of the property. A simplified floor planmay be determined based on the structural data. Structural remodelinginformation may be learned using a learning network applied to the floorplan, simplified floor plan, and structural data. The remodeling planmay be generated by processing the structural remodeling information.

In yet another aspect, embodiments of artificial intelligence systems,methods, computer-readable medium for generating a furnishing plan for aproperty are disclosed. In some embodiments, structural data may beobtained based on a floor plan of the property. Furnishing informationmay be learned by applying the neural network model to the floor planand the structural data. The furnishing information identify one or morefurnishing objects, positions of the respective furnishing objectsplaced in the floor plan, and dimensions of the respective furnishingobjects. The furnishing plan may be generated for the property based onthe furnishing information.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic diagram of an exemplary three-dimensionalview of a real state property, according to embodiments of thedisclosure.

FIG. 2 illustrates an exemplary artificial intelligence system forinterior design, according to embodiments of the disclosure.

FIG. 3 is a block diagram of an exemplary interior design device,according to embodiments of the disclosure.

FIG. 4 illustrates an exemplary user device, according to embodiments ofthe disclosure.

FIG. 5 is an exemplary image of an interior space showing furnishingobjects, according to embodiments of the disclosure.

FIG. 6 is a flowchart of an exemplary method for visualizing furnishingobjects in an image of an interior space, according to embodiments ofthe disclosure.

FIG. 7 is a flowchart of an exemplary method for removing existingfurnishing objects in an image of an interior space, according toembodiments of the disclosure.

FIG. 8 is a flowchart of an exemplary method for suggesting newfurnishing objects for an interior space, according to embodiments ofthe disclosure.

FIG. 9 is a flowchart of an exemplary method for training a neuralnetwork for learning remodeling information for a property, according toembodiments of the disclosure.

FIG. 10 is a flowchart of an exemplary method for generating aremodeling plan for a property, according to embodiments of thedisclosure.

FIG. 11 is a flowchart of an exemplary method for training a neuralnetwork for learning furnishing information for a property, according toembodiments of the disclosure.

FIG. 12 is a flowchart of an exemplary method for generating afurnishing plan for a property, according to embodiments of thedisclosure.

FIG. 13 is a flowchart of an exemplary method for generating a displaymodel visualizing a furnishing plan for a property, according toembodiments of the disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the exemplary embodiments,examples of which are illustrated in the accompanying drawings. Whereverpossible, the same reference numbers will be used throughout thedrawings to refer to the same or like parts.

FIG. 1 illustrates a schematic diagram of an exemplary three-dimensional(3D) view of a real estate property 100 (hereafter “property 100”),according to embodiments of the disclosure. In some embodiments,property 100 may be a residential property such as a house, anapartment, a townhouse, a garage, or a commercial property such as awarehouse, an office building, a hotel, a museum, and a store, etc. Asshown in FIG. 1, the three-dimensional view virtually recreates property100 including its layout (e.g., the framing structures that divide theproperty into several rooms such as walls and counters), finishing(e.g., kitchen/bathroom cabinets, bathtub, island, etc.), fixturesinstalled (e.g., appliances, window treatments, chandeliers, etc.), andfurniture and decorations (e.g., beds, desks, tables and chairs, sofas,TV stands, bookshelves, wall paintings, mirrors, plants, etc.)

In some embodiments, property 100 may include multiple rooms orfunctional spaces separated by interior walls. For example, property 100may include a living room, bedroom, dining room, kitchen, bathroom, etc.As shown in FIG. 1, property 100 may include a great room 110 that hascombined functions of a living room and a kitchen and bedrooms 120 and130.

The three-dimensional view of property 100 may be rendered from multiplepoint clouds acquired of the property. The multiple point clouds may beacquired at different view angles. The point clouds are thenpost-processed and merged to render the 3D view. Consistent with thepresent disclosure, a point cloud is a set of data points in space,which measures the external surface of an object. Point cloud istypically represented by a set of vectors in a three-dimensionalcoordinate system. In some embodiments, the point cloud may include thethree-dimensional coordinates of each data point therein. Point cloudsare generally acquired by 3D scanners, which survey the external surfacesurrounding the object.

Consistent with embodiments of present disclosure, interior design ofproperty 100 may include the remodeling of its structure. For example,as shown in FIG. 1, remodeling great room 110 may include changing theframing structures that separate great room 110 from bedroom 130, orchanging the layout of the kitchen area. Remodeling of great room 110may further include removing or adding windows or doors to the walls.For instances, property owners may typically remodel their propertiesbefore putting them on market for sale or rent or after purchasing itfrom previous owners.

Consistent with embodiments of present disclosure, interior design ofproperty 100 may additionally include furnishing and decorating theinterior space of the property. For example, as shown in FIG. 1, greatroom 110 may be furnished with dining table set 113, a TV stand 114, anda living room set 115. Great room 110 may be further decorated with,e.g., plants 116. Similarly, bedroom 130 may be furnished with a bed 131and a rocking chair 133, and decorated with pictures 132. Sometimes,property owners may want to refurnish/redecorate the respective spaces,to accommodate different use or style. For example, bedroom 130 may beconverted to a nursery in expectation of a newborn, so that bed 130 maybe replaced with a crib and a changing table, and the room may bedecorated with a cartoon theme. As another example, the property ownermay have a change of taste and would like to replace European stylefurniture with modern furniture. Sometimes, properties may be stagedwith staging furniture and decorative pieces before conducting openhouses.

Remodeling or furnishing/refurnishing a property, or a part of theproperty, is a time consuming and high cost project. Property owners donot want to wait until it is completed to find that it is not quite theeffect they have imagined and desired. It would be a hassle to make anyadjustment afterwards. For example, when a piece of furniture ispurchased and delivered, it is difficult to return or change it. It iseven more difficult to undo any structural changes made to the property.The present disclosure provides systems and methods artificialintelligence systems and methods for generating an interior design planfor a space (e.g., property 100) and providing a visualization of thesame, so that the user (e.g., a property owner or an interior designer)could have a close-to-reality feel of the design effect in the space. Insome embodiments, the disclosed systems and methods may use neuralnetworks to intelligently generate the design plans based on theattributes of the actual space and generate the visual representationsbased on the design plans.

FIG. 2 illustrates an exemplary artificial intelligence interior designsystem 200 (referred to as “system 200” hereafter), according to someembodiments of the disclosure. In some embodiments, system 200 may beconfigured to provide interior design suggestions and/or visualrepresentations for an actual space. For example, system 200 may providefurnishing/decoration suggestions based on an image depicting aninterior space provided by the user. As another example, system 200 mayprovide remodeling or furnishing suggestions based on a floor planprovided by the user.

In some embodiments, system 200 may make the design suggestions using amachine learning network. As shown in FIG. 2, system 200 may includecomponents for performing two phases, a training phase and a learningphase. To perform the training phase, system 200 may include a trainingdatabase 201 for storing training data 210 and a model training device202 for training learning models 212. In some embodiments, learningmodels 212 may include learning models for making design suggestions andlearning models for generating visual representations. To perform thelearning phase, system 200 may include interior design device 203 forintelligently generate design plans/suggestions and visualrepresentations using trained learning models 212. In some embodiments,system 200 may include more or less of the components shown in FIG. 2.For example, when learning models 212 are pre-trained and provided,system 200 may include only device 203.

In some embodiments, system 200 may optionally include a network 206 tofacilitate the communication among the various components of system 200,such as databases 201, and devices 202, 203, user device 204, and 3Dscanner 205. For example, network 206 may be a local area network (LAN),a wireless network, a cloud computing environment (e.g., software as aservice, platform as a service, infrastructure as a service), aclient-server, a wide area network (WAN), etc. In some embodiments,network 206 may be replaced by wired data communication systems ordevices.

In some embodiments, the various components of system 200 may be remotefrom each other or in different locations, and be connected throughnetwork 206 as shown in FIG. 2. In some alternative embodiments, certaincomponents of system 200 may be located on the same site or inside onedevice. For example, training database 201 may be located on-site withor be part of model training device 202. As another example, modeltraining device 202 and interior design device 203 may be inside thesame computer or processing device.

As shown in FIG. 2, model training device 202 may communicate withtraining database 201 to receive one or more sets of training data 210.Model training device 202 may use the training data received fromtraining database 201 to train a plurality of learning models (e.g.,trained learning models 212). Trained learning models 212 may includelearning models for learning furnishing information and generatingfurnishing plans, and learning model for learning remodeling informationand generating remodeling plans, and the like. Learning models 212 maybe trained using training data 210 stored in training database 201.

In some embodiments, the training phase may be performed “online” or“offline.” An “online” training refers to performing the training phasecontemporarily with the learning phase. An “online” training may havethe benefit to obtain a most updated learning models based on thetraining data that is then available. However, an “online” training maybe computational costive to perform and may not always be possible ifthe training data is large and/or the models are complicate. Consistentwith the present disclosure, an “offline” training is used where thetraining phase is performed separately from the learning phase. Learnedmodels 212 may be trained offline and saved and reused for assistinginterior design.

Model training device 202 may be implemented with hardware speciallyprogrammed by software that performs the training process. For example,model training device 202 may include a processor and a non-transitorycomputer-readable medium. The processor may conduct the training byperforming instructions of a training process stored in thecomputer-readable medium. Model training device 202 may additionallyinclude input and output interfaces to communicate with trainingdatabase 201, network 206, and/or a user interface (not shown). The userinterface may be used for selecting sets of training data, adjusting oneor more parameters of the training process, selecting or modifying aframework of the learning model, and/or manually or semi-automaticallyproviding ground-truth associated with training data 210.

Trained learning models 212 may be used by interior design device 203 tomake design suggestions to new interior spaces or floor plans. Interiordesign device 203 may receive trained learning models 212 from modeltraining device 202. Interior design device 203 may include a processorand a non-transitory computer-readable medium (discussed in detail inconnection with FIG. 3). The processor may perform instructions of asequence of interior design processes stored in the medium. Interiordesign device 203 may additionally include input and output interfacesto communicate with user device 204, 3D scanner 205, network 206, and/ora user interface (not shown). The user interface may be used forreceiving an image 214 or a floor plan 216 for interior designsuggestions or visualization. The user interface may further provide thedesign plans/suggestions along with the visual representations to userdevice 204 for display.

In some embodiments, user device 204 may be a cellular device or a smartphone, a personal digital assistant (PDA), a laptop computer, a tabletdevice and a wearable device, which may provide network connection andprocess resources to communicate with interior design device 203 throughnetwork 206. User device 204 may also include, for example, an on-boardcomputing system or customized hardware. User device 204 may also rundesignated service applications such as interior design applications toprovide design assistance and suggestions to the user.

User device 204 may include an interface for user interaction. Forexample, the interface may be a touchscreen or a keyboard (physicalkeyboard or soft keyboard) for the user to input data to user device204. In some embodiments of the present disclosure, user may send image214 captured by 3D scanner 205 or floor plan 216 to interior designdevice 203, via user device 204. In some embodiments of the presentdisclosure, interior design device 203 may provide design suggestionsand visual representations to user device 204. User device 204 maydisplay the suggestions and representations to the user through theinterface. For example, user device 204 may display a rendered view ofthe user provided interior space with suggested furniture anddecorations inserted in.

System 200 may further include a 3D scanner 205 to capture depth imagesof an interior space (e.g., a room in property 100). Consistent with thepresent disclosure, 3D scanner 205 may be selected from RGB-D devices,2D/3D LiDARs, stereo cameras, time-of-flight (ToF) cameras, etc. Each ofthese 3D scanners may acquire depth information as well as colorinformation. In some embodiments, 3D scanner 205 may be integrated withuser device 204, e.g., embedded on the back of user device 204. In someembodiments, 3D scanner 205 may be external to user device 204 butconnected to user device 204 via network 206 to transmit the capturedimages to user device 204. In some embodiments, the captured depth image(e.g., image 214) may be sent to and stored on user device 204 first andthe user gets to decide whether and when to send it to interior designdevice 203. In some other embodiments, 3D scanner 205 may send image 214directly to interior design device 203.

In some embodiments, 3D scanner 205 may acquire depth images atdifferent view angles, and point clouds can be determined based on therespective depth images acquired at the respective different viewangles. A depth image is an image or image channel that includes depthinformation between the view point (where 3D scanner 205 is located) andthe surface of the object. The depth image is similar to a grayscaleimage, where each pixel value represents the distance (L) between theacquisition device and the target point on the object surface. Eachpixel value of the depth image occupies a “short” length in storage,which equals to two bytes or 16 bits. For example, the unit length fordistance L may be 1/5000 meters (0.2 millimeters). In that case, onemeter in distance can encompass 13 pixels and a 16-bit storage can store65535 pixel values. It is contemplated that the unit can be selected tobe a different length, as long it is sufficient to differentiate targetpoints in the depth image as well as not introducing burdensomecomputational complexity. The goal is to achieve a balance between thevisual effect and the computational cost.

FIG. 3 is a block diagram of an exemplary interior design device 203,according to embodiments of the disclosure. In some embodiments,interior design device 203 may be implemented by a physical server or aservice in the cloud. In some other embodiments, interior design device203 may be implemented by a computer or a consumer electronic devicesuch as a mobile phone, a pad, or a wearable device. As shown in FIG. 3,interior design device 203 may include a communication interface 302, aprocessor 304, a memory 306, a storage 308, and a bus 310. In someembodiments, interior design device 203 may have different modules in asingle device, such as an integrated circuit (IC) chip (implemented asan application-specific integrated circuit (ASIC) or afield-programmable gate array (FPGA)), or separate devices withdedicated functions. Components of interior design device 203 may be inan integrated device, or distributed at different locations butcommunicate with each other through a network (not shown). The variouscomponents of interior design device 203 may be connected to andcommunicate with each other through bus 310.

Communication interface 302 may send data to and receive data fromcomponents such as user device 204 and 3D scanner 205 via directcommunication links, a Wireless Local Area Network (WLAN), a Wide AreaNetwork (WAN), wireless communication networks using radio waves, acellular network, and/or a local wireless network (e.g., Bluetooth™ orWiFi), or other communication methods. In some embodiments,communication interface 302 can be an integrated services digitalnetwork (ISDN) card, cable modem, satellite modem, or a modem to providea data communication connection. As another example, communicationinterface 302 can be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links can also beimplemented by communication interface 302. In such an implementation,communication interface 302 can send and receive electrical,electromagnetic or optical signals that carry digital data streamsrepresenting various types of information via a network.

Consistent with some embodiments, communication interface 302 mayreceive depth images (e.g., image 214) acquired by 3D scanner 205. Insome embodiments, communication interface 302 may further receive floorplan 216 and other design preferences provided by the user via userdevice 204. In some further embodiments, communication interface 302 mayalso receive trained learning models 212 from model training device 202.Communication interface 302 may provide the received information or datato memory 306 and/or storage 308 for storage or to processor 304 forprocessing.

Processor 304 may include any appropriate type of general-purpose orspecial-purpose microprocessor, digital signal processor, ormicrocontroller. Processor 304 may be configured as a separate processormodule dedicated to interior design. Alternatively, processor 304 may beconfigured as a shared processor module for performing other functionsrelated to or unrelated to interior design. For example, the interiordesign application is just one application installed on a versatiledevice.

As shown in FIG. 3, processor 304 may include multiple modules, such asa furnishing unit 340, a remodeling unit 342, and a view rendering unit344, and the like. These modules (and any corresponding sub-modules orsub-units) can be hardware units (e.g., portions of an integratedcircuit) of processor 304 designed for use with other components or toexecute part of a program. The program may be stored on acomputer-readable medium (e.g., memory 306 and/or storage 308), and whenexecuted by processor 304, it may perform one or more functions.Although FIG. 3 shows units 340-344 all within one processor 304, it iscontemplated that these units may be distributed among multipleprocessors located near or remotely with each other.

Memory 306 and storage 308 may include any appropriate type of massstorage provided to store any type of information that processor 304 mayneed to operate. Memory 306 and storage 308 may be a volatile ornon-volatile, magnetic, semiconductor, tape, optical, removable,non-removable, or other type of storage device or tangible (i.e.,non-transitory) computer-readable medium including, but not limited to,a ROM, a flash memory, a dynamic RAM, and a static RAM. Memory 306and/or storage 308 may be configured to store one or more computerprograms that may be executed by processor 304 to perform imageprocessing, interior design suggestion, and view rendering as disclosedherein. For example, memory 306 and/or storage 308 may be configured tostore program(s) that may be executed by processor 304 to suggestfurniture pieces for an actual space depicted in a user provided image,and then render a view that shows the suggested or user selectedfurniture pieces in the actual space.

Memory 306 and/or storage 308 may be further configured to storeinformation and data used by processor 304. For instance, memory 306and/or storage 308 may be configured to store various data received byinterior design device 203, such as image 214, floor plan 216, userpreference data, and trained learning models 212. Memory 306 and/orstorage 308 may also be configured to store intermediate data generatedby processor 304, such as point cloud data of various objects in image214, attributes of furnishing objects selected for a space, structuraldata, furnishing information or remodeling information learned usinglearning models, remodeling or furnishing plans generated for a floorplan, and views rendered to visualize the remodeled orfurnished/refurnished space. The various types of data may be storedpermanently, removed periodically, or disregarded immediately after itis processed.

FIG. 4 illustrates an exemplary user device 204, according toembodiments of the disclosure. In some embodiments, user device 204 mayinclude an integrated camera 410 for capturing depth images. Forexample, camera 410 may be 3D scanner 205 described above. In someembodiments, display 420 may further function as a user interface toreceive user input. As nonlimiting examples, display 420 may be a LiquidCrystal Display (LCD), a Light Emitting Diode Display (LED), a plasmadisplay, or any other type of display. In some embodiments, user device204 may further include a display 420 for displaying the captured image.Display 420 may include a number of different types of materials, suchas plastic or glass, and may be touch-sensitive to receive commands fromthe user. For example, the display may include a touch-sensitivematerial that is substantially rigid, such as Gorilla Glass™, orsubstantially pliable, such as Willow Glass™.

In some embodiments, display 420 may provide a Graphical User Interface(GUI) 422 presented on the display for user input and data display. Forexample, GUI 422 may display a soft button 424, for the user to press tocapture the depth images. The user may hold user device 204 up so thatcamera 410 captures a view of property 100, and the view may be shown ondisplay 420. When the view shows the desired orientation and objects,the user can press soft button 424 to capture the view as an image. Forexample, FIG. 5 is an exemplary image 500 of an interior space showingfurnishing objects 510-560, according to embodiments of the disclosure.The interior space depicted by image 500 may be part of great room 110in property 100 as shown in FIG. 1. Consistent with the presentdisclosure, furnishing objects may broadly include pieces of furnitureand decorative items. As shown in FIG. 5, image 500 includes furnishingobjects such as a couch 510, a coffee table 520 with a vase 522 placedthereon, a pair of ottomans 530 and 540, a console table 550 withdecorative bottles 552 placed thereon, and pictures 560 hang on thewall.

User device 204 may further include various physical buttons forreceiving different user inputs. For example, physical button 530 may bea home button, when pressed, return to the main menu where variousapplications are displayed. The user may select an application (e.g., aninterior design application) to start the interior design process.

Interior design device 203, along with user device 204 and 3D scanner205 (either integrated as camera 410 or external to user device 204),may be configured to perform methods for generating interior designplans/suggestions and visualizing such designs, such as those shown byflowcharts of FIGS. 6-12.

FIG. 6 is a flowchart of an exemplary method 600 for visualizingfurnishing objects in an image of an interior space, according toembodiments of the disclosure. In some embodiments, method 600 may beperformed by processor 304 of interior design device 203, e.g.,furnishing unit 340 and view rendering unit 344. Method 600 may includesteps 602-610 as described below. It is to be appreciated that some ofthe steps may be optional to perform the disclosure provided herein.Further, some of the steps may be performed simultaneously, or in adifferent order than shown in FIG. 6. For description purpose, method600 will be described as to visualize image 500 and furnishing objectstherein (as shown in FIG. 5). Method 600, however, can be implementedfor visualizing furnishing of other spaces of a property.

In step 602, interior design device 203 may receive an image of aninterior space with furnishing objects, such as image 500 withfurnishing objects 510-560 as shown in FIG. 5. The image may be capturedby 3D scanner 205 or camera 410 and may be selected to be sent by a uservia user device 204. In some embodiments, the image may be a depth imagethat preserves depth information of the interior space.

In step 604, furnishing unit 340 may remove existing furnishing objectsfrom the image. For example, some or all of furnishing objects 510-560may be removed from image 500. In some embodiments, furnishing unit 340may use object detection methods (such as convolutional neural network(CNN) based detection methods) to first detect the furnishing objects inthe image, and then delete the corresponding pixels of those furnishingobjects from the image. The CNN model may be part of learning models 212trained by model training device 202. Deleting pixels may be implementedby replacing the pixel values with a predetermined value, such as 0. Asa result of removing the furnishing objects, a blank region (or referredto a hole) where the furnishing objects used to occupy may be left inthe image. The blank region defines the contour of the furnishingobjects.

In some embodiments, FIG. 7 is a flowchart of an exemplary method 700for removing existing furnishing objects in an image of an interiorspace, according to embodiments of the disclosure. In some embodiments,method 700 may also be implemented by furnishing unit 340 to performstep 604 of method 600. Method 700 may include steps 702-708 asdescribed below. It is to be appreciated that some of the steps may beoptional to perform the disclosure provided herein. Further, some of thesteps may be performed simultaneously, or in a different order thanshown in FIG. 7.

In step 702, furnishing unit 340 may determine 3D point cloud data ofthe image based on the depth information. In some embodiments, image 400may be a depth image with a channel of depth information. The depthinformation indicates the distance between the view point (e.g., theposition of 3D scanner 205) and each surface point the object beingcaptured. Furnishing unit 340 may determine the 3D point cloud data ofthe image based on such distances. Each point cloud data point may berepresented by a set of 3D coordinates.

In step 704, furnishing unit 340 may identify target point cloud data ofthe furnishing objects in the image. In some embodiments, the 3D pointcloud data of the image may be segmented into several subsets, eachcorresponding to a furnishing object. For example, point cloud datacorresponding to couch 510, coffee table 520, and pair of ottomans 530and 540 may be segmented from the 3D point cloud data. Based on theuser's selection of one or more furnishing objects, furnishing unit 340may then select the corresponding subsets of point cloud data as thetarget point cloud data.

In step 706, furnishing unit 340 may determine positions of thefurnishing objects in the image based on the target point cloud data.For example, positions of couch 510, coffee table 520, and pair ofottomans 530 and 540 may be determined based on the target point clouddata. In some embodiments, furnishing unit 340 may determine thecontours of the furnishing objects using the target point cloud data,which define their positions. In some embodiments, contours of thefurnishing objects may be learned using a learning model. The learningmodel may be part of learning models 212 trained by model trainingdevice 202 using training point cloud data of furnishing objects andtheir corresponding contour labels. For example, the training images mayinclude various different styled or dimensioned couches and theircontour labels, and the trained learning model may accordingly used tolearn the contour of couch 510 from image 500.

In step 708, furnishing unit 340 may remove the furnishing objects fromthe image based on their determined positions. In some embodiments,furnishing unit 340 may reset the values of pixels within the determinedcontours to a predetermined value, e.g., 0 (corresponding to whitecolor).

Returning to FIG. 6, in step 606, furnishing unit 340 may restore theimage. In some embodiments, the image is restored by filling the blankregion with a scene of the interior space that was previously blocked bythe furnishing objects. For example, after removing couch 510 andconsole table 550 from image 500, the blank region left may be filledwith the scene of walls, windows and curtains that were blocked by theremoved objects. The blank region left by removing coffee table 520 andpair of ottomans 530 and 540 may be filled with a floor or carpetconsistent with the flooring otherwise shown in the image.

In some embodiments, a neural network may be used to restore the image.In some embodiments, furnishing unit 340 may input the image obtained bystep 604 into the trained neural network to obtain the restored image.The neural network extracts features from regions outside the blankregion in the image to learn features in the blank region. The neuralnetwork for restoration may be part of learning models 212 trained bymodel training device 202. For example, the neural network may betrained using image inpainting algorithms based on pairs of sampleimages each including a furniture object removed image and itscorresponding restored image. In some embodiments, the neural networkcan be trained using a gradient-decent method or a back-propagationmethod, which gradually adjust the network parameters to minimize thedifference between the restored image output by the network and theground-truth restored image provided as part of the training data. Thetraining may be an iterative process ending upon at least one of thefollowing conditions is satisfied: (1) training time exceeds apredetermined time length; (2) number of iterations exceed apredetermined iteration threshold; (3) a loss function (e.g., across-entropy loss function) calculated based on the restored imageoutput by the network and the ground-truth restored image is smallerthan a predetermined loss threshold. It is contemplated that thetraining may be performed “on-line” or “off-line.”

In step 608, furnishing unit 340 may obtain and insert new furnishingobjects into target positions of the restored image. In someembodiments, the target positions are where the positions of the removedfurnishing objects. The target positions may be represented bycoordinates in a coordinate system constructed using the center of theimage as the origin. The new furnishing objects may be obtained from alocal storage such as memory 306 or storage 308, or obtained remotelyfrom a storage device via network 206. In some embodiments, the newfurnishing objects may be selected or provided by the user via userdevice 204. In some embodiments, the new furnishing objects may beautomatically selected by furnishing unit 340 and suggested to the user,as will be described in connection with FIG. 8 of this disclosure.

In step 610, furnishing unit 340 may adjust the inserted new furnishingobjects in the restored image. In some embodiments, the dimensions ofthe new furnishing objects may be adjusted to target dimensions. Forexample, the target dimensions may be determined based on the size andshape of the removed furnishing objects. As a result, the inserted newfurnishing objects may fit into the area where the removed furnishingobjects previously occupied. Specifically, furnishing unit 340 mayconstruct a 3D model for the new furnishing object based on itsdimensions determined based on the point cloud data of the furnishingobject. Furnishing unit 340 then adjusts the size of the 3D model to fitit into the “hole” left from removing the furnishing objects in the 3Dpoint cloud data. Accordingly, the target dimensions of the newfurnishing object may conform to the 3D dimensions of the target pointcloud data of the removed furnishing objects.

In some other embodiments, the target dimensions may be determined basedon the ratio between the physical dimensions of the removed furnishingpieces and the physical dimensions of the space captured by the image.For example, the height of the inserted new furnishing object is 1meter, and the height of the imaged space is 3 meters, the inserted newfurnishing object may be adjusted to be ⅓ of the size of the image.

View rendering unit 344 may render a 3D view of the space with the newfurnishing objects and send the view to user device 204 for display. Byadjusting the dimensions of the inserted furnishing objects, thefurnishing objects may blend well in the imaged space. Accordingly, thevisualization of the refurnished space may be closer to reality.

FIG. 8 is a flowchart of an exemplary method 800 for suggesting newfurnishing objects for an interior space, according to embodiments ofthe disclosure. In some embodiments, method 800 may be performed byprocessor 304 of interior design device 203, e.g., furnishing unit 340and view rendering unit 344. Method 800 may include steps 802-818 asdescribed below. It is to be appreciated that some of the steps may beoptional to perform the disclosure provided herein. Further, some of thesteps may be performed simultaneously, or in a different order thanshown in FIG. 8. For description purpose, method 600 will also bedescribed as to furnishing the space depicted by image 500 (as shown inFIG. 5). Method 800, however, can be implemented for furnishing otherspaces of a property.

In step 802, interior design device 203 may receive an image of aninterior space with furnishing objects, such as image 500 withfurnishing objects 510-560 as shown in FIG. 5, similar to step 602.

In step 804, furnishing unit 340 may determine feature information offurnishing objects in the image using a learning model. For example,feature information of furnishing objects 510-560 in image 500 may belearned. Consistent with the disclosure, feature information may befeatures that define the furnishing objects, such as their categories,styles, dimensions, and functions, etc. In some embodiments, furnishingunit 340 may use object detection methods (such as neural network basedobject detectors) to detect the furnishing objects and their features.

The neural network may learn the mapping between images and features offurnishing objects. In some embodiments, the neural network may betrained by model training device 202 using a Single Shot MultiBoxDetector (SSD) or a Deformable Part Model (DPM) as an initial model. Theinitial model is then adjusted (by adjusting the model parameters)during training. The neural network may be trained using sample imagesand ground-truth object features. During each iteration of the trainingprocess, the training images are input into the model, and the outputfeatures from the model are compared with the ground-truth objectfeatures. The model parameters are adjusted based on a differencebetween the two. The training ends upon satisfying at least one of thefollowing stopping criteria: (1) training time exceeds a predeterminedtime length; (2) number of iterations exceed a predetermined iterationthreshold; (3) a loss function (e.g., a cross-entropy loss function)calculated based on the output features from the model and theground-truth object features is smaller than a predetermined lossthreshold. It is contemplated that the training may be performed“on-line” or “off-line.”

In step 806, furnishing unit 340 determines 3D point cloud data of theimage based on the depth information. In some embodiments, image 400 maybe a depth image with a channel of depth information. The depthinformation indicates the distance between the view point (e.g., theposition of 3D scanner 205) and each surface point the object beingcaptured. Furnishing unit 340 may determine the 3D point cloud data ofthe image based on such distances. Each point cloud data point may berepresented by a set of 3D coordinates.

In step 808, furnishing unit 340 determines dimension information offurnishing objects in the image based on the 3D point cloud data. Insome embodiments, the 3D point cloud data may be segmented into severalsubsets, each corresponding to a furnishing object. Furnishing unit 340may determine the dimension information of each furnishing objects basedon the 3D coordinates of the data points within the corresponding subsetof point cloud data.

In step 810, furnishing unit 340 may determine attributes of thefurnishing objects based on the feature information determined in step804 and/or dimension information determined in step 808. In someembodiments, the attributes may be the feature information, thedimension information, or a combination of both. In some embodiments,the attributes may further include other information of the furnishingobject, such as model number, product number (e.g., UPC), and productname, etc.

In step 812, furnishing unit 340 may select target furnishing objectsthat do not match with the interior space based on their attributes. Insome embodiments, furnishing unit 340 may determine whether a furnishingobject matches with the interior space based on style. As described instep 810, style of the furnishing object may be part of its featureinformation included as its attributes. Exemplary styles of a furnishingobject may include European style, oriental style, contemporary style,modern style, etc. In some embodiments, the style of a furnishing objectcan be learned by the object detection learning network described instep 804. For example, couch 510, coffee table 520, and pair of ottomans530 and 540 may be all contemporary style. Style of the interior spacemay be defined collectively by the styles of furnishing objects withinthat space. For example, if most the furnishing objects are orientalstyle, the interior space is determined to be oriental style. Becausefurnishing objects 510-540 are all contemporary style, the interiorspace depicted by image 500 therefore is also contemporary style.Furnishing unit 340 may then compare the style of each furnishing objectand the style of the interior space to determine whether the match. If afurnishing object is oriental style but the interior space iscontemporary style, the furnishing object is identified as a targetfurnishing object that does not match.

In some embodiments, furnishing unit 340 may determine whether afurnishing object matches with the interior space based on itsdimensions. As described in step 810, dimension information of thefurnishing object may also be included as its attributes. In someembodiments, if the dimensions of a furnishing object are larger thanthe unoccupied size of the interior space, the furnishing object can bedetermined as a target furnishing object that does not match theinterior space. In some alternative embodiments, furnishing unit 340 mayconsider the combination of feature information (e.g., style) anddimension information when selecting mismatched furnishing objects.

In step 814, furnishing unit 340 may generate an indication the targetobjects do not match with the interior space depicted in the image. Insome embodiments, the indication can be in the form of at least one ofan image, a text, an audial signal, etc. For example, the indication maybe a text message that “The furniture style does not match with theroom. Please consider replace it.” As another example, the indicationmay be an image of the room with the mismatched furniture highlighted.As yet another example, the indication may be a voice messageidentifying the mismatched furniture. In some embodiments, theindication may include more than one form, such as an image paired witha text message. The indication may be sent to user device 204 fordisplay to the user.

In step 816, furnishing unit 340 may select object information from itsobject database based on the attributes. Consistent with the presentdisclosure, object information may include at least one of object name,category, image, and place of origin, etc. For example, the objectinformation may include the feature information of furnishing objects.In some embodiments, the object database may be stored locally ininterior design device 203, e.g., in memory 306 or storage 308. In someother embodiments, the object database may be stored remotely frominterior design device 203, e.g., in the cloud.

In some embodiments, the object information may be selected according tothe style of the interior space determined as in step 812. For example,the object information may be the feature information of furnishingobjects that have the same style as that of the interior space. As oneexample, if the interior space is oriental style, furnishing unit 340may select furnishing objects of oriental style and select the otherfeature information (e.g., object name, category, image, and place oforigin) of those furnishing objects as the object information.

In some embodiments, the object information may be selected according tothe dimensions of the interior space. For example, the objectinformation may be the feature information of furnishing objects thathave the right sizes that fit within the dimensions of the interiorspace. In some embodiments, furnishing unit 340 may determine a range ofobject dimensions based on the image dimensions and predetermined firstratio (used to determine the lower limit of the range) and second ratio(used to determine the upper limit of the range). Object information offurnishing objects that have dimensions falling in the range is selectedby furnishing unit 340. In some alternative embodiments, furnishing unit340 may consider the combination of feature information (e.g., style)and dimension information when selecting object information.

In step 818, furnishing unit 340 may generate a suggestion according tothe object information. In some embodiments, the indication can be inthe form of at least one of an image, a text, an audial signal, etc. Forexample, the indication may be a text message showing the objectinformation (e.g., object name, category, image, and place of origin).As another example, the indication may be an image of the room withsuggested furnishing objects that satisfy the object information.Furnishing unit 340 may identify suitable furnishing objects based onthe object information and replace the mismatched furnishing objectswith the new furnishing objects in the image by performing, e.g., method600. As yet another example, the indication may be a voice messageexplaining the object information. In some embodiments, the indicationmay include more than one form, such as an image paired with a textmessage. The suggestion may be sent to user device 204 for display tothe user.

FIG. 9 is a flowchart of an exemplary method 900 for training a neuralnetwork for learning remodeling information for a property, according toembodiments of the disclosure. In some embodiments, method 900 may beperformed by model training device 202. Method 900 may include steps902-910 as described below. It is to be appreciated that some of thesteps may be optional to perform the disclosure provided herein.Further, some of the steps may be performed simultaneously, or in adifferent order than shown in FIG. 9.

In step 902, model training device 202 receives sample floor plans andcorresponding sample remodeling data. For example, model training device202 may receive such training data from training database 201. A floorplan may be a drawing that describes the structure and layout of aproperty. For example, the floor plan may describe the structures thatdivide property 100 into different functional rooms 110-130, and thedetailed shape and dimensions of each room. In some embodiments, thesample floor plans may be generated by Computer-Aided Design (CAD)modeling tools. Each sample floor plan may show structures such aswalls, counters, stairs, windows, and doors, etc. In some embodiments,the sample floor plan may be a vector graph.

The corresponding sample remodeling data may include structuralremodeling information. In some embodiments, the structural remodelingmay include, e.g., to knock down a wall, to reduce a wall to a halfwall, to add a wall, to move a wall to a different place, toremove/expand/insert a window, or to remove/insert a door, etc.Accordingly, the sample remodeling data may include the identity of eachstructure for remodeling, and descriptions of the remodeling, including,e.g., position of the structure, dimensions of the structure before andafter the remodeling, etc. In some embodiments, the structuralremodeling information may further include a structural heat map. Thestructural heat map reflects the structures (e.g., walls, windows,doors, etc.) for remodeling and the respective probabilities thestructures need to be remodeled.

In step 904, model training device 202 may obtain sample structural datacorresponding to the sample floor plan. In some embodiments, the samplefloor plan may contain corresponding structural data including, e.g.,wall distribution data, weight-bearing wall distribution data,window/door distribution data, area data, ceiling height data, structureposition coordinates, etc.

In step 906, model training device 202 may annotate the samplestructural data to add information, such as floor plan featureinformation and/or grading information. In some embodiments, the floorplan feature information may include, e.g., spaciousness, number ofoccupants, storage space, lighting condition, year the property wasbuilt, etc. In some embodiments, the sample structural data may begraded to generate the grading information. For example, theweight-bearing wall distribution or the window/door distribution may begraded. In some embodiments, the grading information may be a number(e.g., on a scale of 0-100) or a grade level (e.g., A-F) indicating thequality of the structural data. The feature information and gradinginformation may be annotated on the respective structure in the samplefloor plan or added to the corresponding sample structural data.

In step 908, model training device 202 may determine a first simplifiedfloor plan based on the annotated structural data. In some embodiments,model training device 202 may identify the weight-bearing walls based onthe sample structural data, in particular, the weight-bearing walldistribution data. Model training device 202 then determines the firstsimplified floor plan according to the weight-bearing walls. Forexample, the first simplified floor plan may only include theweight-bearing walls, as well as windows and doors on thoseweight-bearing walls. If there is no weight-bearing wall in the samplefloor plan, the first simplified floor plan may be set as the originallyreceived sample floor plan.

In step 910, model training device 202 may train a neural network forlearning remodeling information. In some embodiments, the neural networkmay be trained using sample floor plans (with their derived samplestructural data and first simplified floor plans) and theircorresponding sample remodeling data. In some embodiments, the neuralnetwork is trained for a remodeling preference. For example, theremodeling preference may be to increase the storage space in theproperty, or to improve the overall living experience (e.g., morecomfortable). Accordingly, the neural network may be trained using aloss function that reflects the remodeling preference. For example, theloss function may be calculated as the collective area of storage spacein the property when the remodeling preference is set to increase thestorage space.

In some embodiments, the training may be an iterative process. At eachiteration, the neural network generates structural remodelinginformation based on the sample floor plan, the structural data and thesimplified floor plan. In some embodiments, the neural network can betrained using a gradient-decent method or a back-propagation method,which gradually adjust the network parameters to minimize the differencebetween the structural remodeling information output by the network andthe ground-truth sample remodeling data provided as part of the trainingdata in step 902. The training may end upon at least one of thefollowing conditions is satisfied: (1) training time exceeds apredetermined time length; (2) number of iterations exceed apredetermined iteration threshold; (3) a loss function (e.g., across-entropy loss function) calculated based on the structuralremodeling information output by the network and the ground-truthremodeling data is smaller than a predetermined loss threshold. It iscontemplated that the training may be performed “on-line” or “off-line.”

In some embodiments, the neural network may adopt any suitablestructure, such as a floornet model. The neural network may include aninput layer, an intermediate layer and an output layer, each layerreceives the output of its previous layer as its input. In someembodiments, the intermediate layer may be a convolution layer. In someembodiments, the intermediate layer may be a fully connected layer.

FIG. 10 is a flowchart of an exemplary method 1000 for generating aremodeling plan for a property, according to embodiments of thedisclosure. In some embodiments, method 1000 may be performed byprocessor 304 of interior design device 203, e.g., remodeling unit 342and view rendering unit 344. Method 1000 may include steps 1002-1014 asdescribed below. It is to be appreciated that some of the steps may beoptional to perform the disclosure provided herein. Further, some of thesteps may be performed simultaneously, or in a different order thanshown in FIG. 10. For description purpose, method 1000 will be describedusing property 100 (as shown in FIG. 1) as an example. Method 1000,however, can be implemented for remodeling other spaces or otherproperties.

In step 1002, remodeling unit 342 receives a floor plan that needsremodeling. For example, remodeling unit 342 may receive floor plan 216provided by the user via user device 204. Floor plan 216 may be adrawing that describes the structure and layout of a property. Forexample, floor plan 216 may describe the structures that divide property100 into different functional rooms 110-130, and the detailed shape anddimensions of each room. Floor plan 216 may show structures such aswalls, counters, stairs, windows, and doors, etc. In some embodiments,floor plan 216 may be a vector graph generated by CAD modeling tools.

In step 1004, remodeling unit 342 may obtain structural datacorresponding to the floor plan. In some embodiments, floor plan 216 maycontain corresponding structural data including, e.g., wall distributiondata, weight-bearing wall distribution data, window/door distributiondata, area data, ceiling height data, structure position coordinates,etc.

In step 1006, remodeling unit 342 may determine a second simplifiedfloor plan based on the structural data. In some embodiments, remodelingunit 342 may identify the weight-bearing walls of the property based onthe structural data, in particular, the weight-bearing wall distributiondata. Remodeling unit 342 then determines the second simplified floorplan according to the weight-bearing walls. For example, the secondsimplified floor plan may only include the weight-bearing walls, as wellas windows and doors on those weight-bearing walls. If there is noweight-bearing wall in the property, the second simplified floor planmay be set as the originally received floor plan.

In step 1008, remodeling unit 342 may learn structural remodelinginformation based on the second simplified floor plan, the originallyreceived floor plan, and the structural data. In some embodiments, thestructural remodeling information may be learned by applying the neuralnetwork trained using method 900. The structural remodeling informationmay include the identity of each structure for remodeling, descriptionsof the remodeling, including, e.g., position of the structure,dimensions of the structure before and after the remodeling, and thecorresponding structural heat map, etc.

In some embodiments, the structural remodeling information is generatedaccording to a remodeling preference, e.g., provided by the user viauser device 204. For example, the remodeling preference may be toincrease the storage space in the property, or to improve the overallliving experience (e.g., more comfortable). Accordingly, a neuralnetwork trained to reflect the remodeling preference may be used in step1008.

In step 1010, remodeling unit 342 may generate a remodeling plan basedon the structural remodeling information. In some embodiments,remodeling unit 342 may process the structural remodeling informationsubject to certain predetermined remodeling decision rules (e.g.,construction regulations) when generating the remodeling plan. In someembodiments, remodeling unit 342 may use the remodeling decision rulesas constraints for optimizing the structural remodeling information. Forexample, a Monte-Carlo search tree (MCST) algorithm may be used.

In some embodiments, the remodeling decision rules may be predetermined.For example, the pivot point of each structure, determined using, e.g.,Integer Programming, may be used as a constraint. Accordingly, aMonte-Carlo search tree may be constructed using the structures in theheat map as tree nodes. The MCST algorithm then traverses the learnedstructural heat map by traversing the tree nodes, in order to identifystructures (e.g., walls, windows, or doors) for remodeling. Theidentified structures have to satisfy all the remodeling decision rulesand maximize the overall remodeling probability of the remodeling plan.The overall remodeling probability of the remodeling plan may be the sumor the weighted sum of respective probabilities (according to the heatmap) of the structures identified for remodeling.

After the structures for remodeling are identified, the remodeling planmay be generated according to the remodeling information. In someembodiments, the remodeling plan may include an adjusted floor plan thatreflects the remodeled property.

In step 1012, remodeling unit 342 may further generate descriptiveinformation that describes the remodeling plan. In some embodiments,remodeling unit 342 may compare the original floor plan the remodelingplan and generate the descriptive information according to thedifference. For example, the descriptive information may describe thestructural changes that should be made, e.g., to knock down a wall, toreduce a wall to a half wall, to add a wall, to move a wall to adifferent place, to remove/expand/insert a window, or to remove/insert adoor, etc. The descriptive information may further include informationrelated to the remodeling projection, such as the construction materialsnecessary for the remodeling, and expected time needed for complete theremodeling.

In step 1014, remodeling unit 342 may send the remodeling plan and thedescriptive information to the user. For example, the remodeling planand the descriptive information may be sent to user device 204 fordisplay.

FIG. 11 is a flowchart of an exemplary method 1100 for training a neuralnetwork for learning furnishing information for a property, according toembodiments of the disclosure. In some embodiments, method 1100 may beperformed by model training device 202. Method 1100 may include steps1102-1110 as described below. It is to be appreciated that some of thesteps may be optional to perform the disclosure provided herein.Further, some of the steps may be performed simultaneously, or in adifferent order than shown in FIG. 11.

In step 1102, model training device 202 receives first sample floorplans. For example, model training device 202 may receive the firstsample floor plans from training database 201. A floor plan may be adrawing that describes the structure and layout of a property. Forexample, the floor plan may describe the structures that divide property100 into different functional rooms 110-130, and the detailed shape anddimensions of each room. In some embodiments, the first sample floorplans may be generated by Computer-Aided Design (CAD) modeling tools.Each sample floor plan may show structures such as walls, counters,stairs, windows, and doors, etc. In some embodiments, the sample floorplan may be a vector graph.

In step 1104, model training device 202 may obtain sample structuraldata corresponding to the first sample floor plans. In some embodiments,the sample floor plan may contain corresponding structural dataincluding, e.g., wall distribution data, weight-bearing walldistribution data, window/door distribution data, area data, ceilingheight data, structure position coordinates, etc.

In step 1106, model training device 202 may acquire second sample floorplans corresponding to the first sample floor plans. In someembodiments, the second sample floor plans may be identical or similarin layout, structure, and size as the corresponding first sample floorplans. The second sample floor plans may be identified using thestructural data of the first sample floor plans. In some embodiments,the first sample floor plans may be furnished or unfurnished, but thecorresponding second sample floor plans are furnished with one or morefurnishing objects. For example, a second sample floor plan may besimilar to what is shown in FIG. 1. The furnishing objects may be piecesof furniture, e.g., dining set 113, TV stand 114, living room set 115,bed 131, and rocking chair 133, or decorative objects, e.g., plants 116and pictures 132.

In step 1108, model training device 202 may label the furnishing objectsin the second sample floor plans to generate sample furnishinginformation. In some embodiments, the furnishing objects may be manuallylabeled. The sample furnishing information may include, e.g., categoryof the furnishing object, position in the floor plan, orientation ofplacement, style, and dimensions, etc. In some embodiments, the samplefurnishing information may further include grading information. Forexample, the grading information may be a number (e.g., on a scale of0-100) or a grade level (e.g., A-F). In some embodiments, the furnishinginformation may further include a furnishing heat map. The furnishingheat map reflects the placement of furnishing objects and the respectiveprobabilities the furnishing objects will be placed at the respectivepositions. For example, the furnishing heat map shows the recommendedplacement of dining set 113 and living room set 115 in great room 110,the probabilities dining set 113 and living room set 115 be placed atthose positions, and other information of dining set 113 and living roomset 115.

In step 1110, model training device 202 may train a neural network forlearning furnishing information. In some embodiments, the neural networkmay be trained using the first sample floor plans and the samplefurnishing information generated form their corresponding second samplefloor plans. In some embodiments, the training may be an iterativeprocess. At each iteration, the neural network generates outputfurnishing information based on the first sample floor plans and theirstructural data. In some embodiments, the neural network can be trainedusing a gradient-decent method or a back-propagation method, whichgradually adjust the network parameters to minimize the differencebetween the furnishing information output by the network when applied tothe first sample floor plans and the sample furnishing informationgenerated from the corresponding second sample floor plans in step 1108.The training may end upon at least one of the following conditions issatisfied: (1) training time exceeds a predetermined time length; (2)number of iterations exceed a predetermined iteration threshold; (3) aloss function (e.g., a cross-entropy loss function) calculated based onthe furnishing information output by the network when applied to thefirst sample floor plans and the sample furnishing information generatedfrom the corresponding second sample floor plans is smaller than apredetermined loss threshold. It is contemplated that the training maybe performed “on-line” or “off-line.”

In some embodiments, the neural network may adopt any suitablestructure, such as a ResNet/DenseNet model. ResNet (Residual NeuralNetwork) can expedite the training as well as improve the learningaccuracy of the network. The neural network may include an input layer,an intermediate layer and an output layer, each layer receives theoutput of its previous layer as its input. In some embodiments, theintermediate layer may be a convolution layer. In some embodiments, theintermediate layer may be a fully connected layer.

FIG. 12 is a flowchart of an exemplary method 1200 for generating afurnishing plan for a property, according to embodiments of thedisclosure. In some embodiments, method 1200 may be performed byprocessor 304 of interior design device 203, e.g., furnishing unit 340.Method 1200 may include steps 1202-1214 as described below. It is to beappreciated that some of the steps may be optional to perform thedisclosure provided herein. Further, some of the steps may be performedsimultaneously, or in a different order than shown in FIG. 12. Fordescription purpose, method 1200 will be described using property 100(as shown in FIG. 1) as an example. Method 1200, however, can beimplemented for furnishing other spaces or other properties.

In step 1202, furnishing unit 340 receives a floor plan. The floor planmay be furnished or unfurnished. For example, furnishing unit 340 mayreceive floor plan 216 provided by the user via user device 204. Floorplan 216 may be a drawing that describes the layout, structure, and sizeof a property. In some embodiments, the floor plan may be a vector graphgenerated by CAD modeling tools.

In step 1204, furnishing unit 340 may obtain structural datacorresponding to the floor plan. In some embodiments, floor plan 216 maycontain corresponding structural data including, e.g., wall distributiondata, weight-bearing wall distribution data, window/door distributiondata, area data, ceiling height data, structure position coordinates,etc.

In step 1206, furnishing unit 340 may learn furnishing information basedon the floor plan and the structural data. In some embodiments, thefurnishing information may be learned by applying the neural networktrained using method 1100. The furnishing information may include, e.g.,category of the furnishing object recommended for placing in the floorplan, position of each furnishing object in the floor plan, orientationof placement, style, and dimensions, etc. In some embodiments, thelearned furnishing information may further include a furnishing heatmap.

In step 1208, furnishing unit 340 may generate a furnishing plan byprocessing the furnishing information. In some embodiments, furnishingunit 340 may process the furnishing information subject to certainpredetermined furnishing decision rules when generating the furnishingplan. In some embodiments, furnishing unit 340 may use the furnishingdecision rules as constraints for optimizing the furnishing plan. Forexample, an MCST algorithm may be used.

In some embodiments, the furnishing decision rules may be predetermined.For example, an energy function E(x) may be constructed based on theplacement positions and the furnishing objects to be placed, and theenergy function min(ΣE(x)) may be used as a constraint. The energyfunction may consider the position of each furnishing object relative tothe walls and relative to other furnishing objects. As another example,the total number of the furnishing objects may be another constraint.Accordingly, a Monte-Carlo search tree may be constructed using theplaced furnishing objects in the heat map as tree nodes. The MCSTalgorithm then traverses the learned furnishing heat map by traversingthe tree nodes, in order to identify the furnishing objects and theirplacement positions in the floor plan. The placement has to satisfy allthe furnishing decision rules and maximize the overall placementprobability of the furnishing plan. The overall placement probability ofthe furnishing plan may be the sum or the weighted sum of respectiveprobabilities (according to the heat map) of the furnishing objects.

After the furnishing objects for placement and their respectiveplacement positions are identified, the furnishing plan may be generatedaccording to the furnishing information. In some embodiments, thefurnishing plan may include a graphic illustration of the furnitureplacement in the floor plan.

In step 1210, furnishing unit 340 may further generate displayinformation that describes the furnishing plan. In some embodiments, thedescriptive information may include the furnishing information of eachfurnishing object selected for placement in the floor plan, e.g.,category of the furnishing object, position of the furnishing object inthe floor plan, orientation of placement, style, and dimensions of thefurnishing object, etc.

In step 1212, furnishing unit 340 may further obtain accessory objectsbased on the display information. The accessory objects may complementthe furnishing objects. For example, the furnishing objects may includebed 131 placed in bedroom 130, and the accessory objects may includepictures 132 to be hung on the wall behind bed 131 and beddings used onbed 131. The accessory objects may further include other pieces offurniture that usually pair with the furnishing object but not yetincluded in the furnishing plan. For example, one or more nightstandsmay be identified to pair with bed 131. In some embodiments, theaccessory objects may be selected consistent with display information ofthe furnishing objects, such as style and dimensions. For example, thenightstands may be selected to be the same style as bed 131 and thebeddings may be selected to fit the size of bed 131 (e.g., king-sized,or queen-sized).

In step 1214, furnishing unit 340 may optimize the placement of thefurnishing objects and the accessory objects in the furnishing planbased on furnishing preferences. In some embodiments, the furnishingpreferences may be provided by the user via user device 204. Thefurnishing preference may be against the wall (i.e., minimum gap betweenthe furnishing object and the wall) or against the floor. Accordingly,the furnishing objects and accessory objects may be moved in thefurnishing plan according to the furnishing preferences.

FIG. 13 is a flowchart of an exemplary method 1300 for generating adisplay model visualizing a furnishing plan for a property, according toembodiments of the disclosure. In some embodiments, method 1300 may beperformed by processor 304 of interior design device 203, e.g., viewrendering unit 344. Method 1300 may include steps 1302-1312 as describedbelow. It is to be appreciated that some of the steps may be optional toperform the disclosure provided herein. Further, some of the steps maybe performed simultaneously, or in a different order than shown in FIG.13.

In step 1302, view rendering unit 344 may generate a 3D property modelbased on the floor plan. In some embodiments, the 3D property model maybe generated using CAD modeling tools based on the structural dataderived from the floor plan. The 3D property model may display a view ofthe structures and layout of an unfurnished property.

In step 1304, view rendering unit 344 may generate 3D object models forthe furnishing objects and the accessory objects in the furnishing planbased on the display information. Similarly, the 3D object models mayalso be generated using CAD modeling tools, based on display informationsuch as category, style, and dimensions.

In step 1306, view rendering unit 344 may fit the 3D object models inthe 3D property model based on the furnishing plan. The 3D object modelsare placed at the respective positions in the 3D property model asspecified by the furnishing information provided by the furnishing plan.

In step 1308, view rendering unit 344 may determine a design style basedon the display information or the furnishing information. In someembodiments, the display information or the furnishing informationspecifies the style of each furnishing objects and accessary objects.Exemplary styles of a furnishing/accessory object may include Europeanstyle, oriental style, contemporary style, modern style, etc. The designstyle of an interior space may be defined collectively by the styles offurnishing objects within that space. For example, if most thefurnishing/accessory objects in great room 110 are oriental style, thedesign style of great room 110 may be determined to be oriental style.In some embodiments, the property may have different design styles indifferent functional spaces. For example, great room 110 may have anoriental style and bedroom 130 may have a contemporary style.

In step 1310, view rendering unit 344 may render a 3D display model ofthe property based on the design style(s). For example, the 3D view ofproperty 100 shown in FIG. 1 may be an example of the 3D display modelrendered in step 1310. In some embodiments, view rendering unit 344 mayadjust the pattern or color of items like window treatments, curtains,tiles, carpets, and hardwood flooring to conform to the respectivedesign styles.

In step 1312, view rendering unit 344 may send the 3D display model tothe user. For example, the 3D display model may be sent to user device204 for display. In some embodiments, the 3D display model may bedisplayed within a Virtual Reality (VR) tour application, which offerstools for the user to navigate through the 3D display model and reviewthe display information associated with the various furnishing/accessoryobjects.

Another aspect of the disclosure is directed to a non-transitorycomputer-readable medium storing instruction which, when executed, causeone or more processors to perform the methods, as discussed above. Thecomputer-readable medium may include volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other typesof computer-readable medium or computer-readable storage devices. Forexample, the computer-readable medium may be the storage device or thememory module having the computer instructions stored thereon, asdisclosed. In some embodiments, the computer-readable medium may be adisc or a flash drive having the computer instructions stored thereon.

Although the embodiments are described using interior design of indoorspaces as examples, it is contemplated that the concepts could bereadily expanded and adapted to design of outdoor spaces, such as thedeck, the front/back yard, the garage, as well as the neighboringenvironment. A person of ordinary skill can adapt the disclosed systemsand methods without undue experimentation for outdoor designs.

It will be apparent to those skilled in the art that variousmodifications and variations can be made to the disclosed system andrelated methods. Other embodiments will be apparent to those skilled inthe art from consideration of the specification and practice of thedisclosed system and related methods.

It is intended that the specification and examples be considered asexemplary only, with a true scope being indicated by the followingclaims and their equivalents.

The invention claimed is:
 1. A system for generating a furnishing planfor a property, comprising: a communication interface configured toreceive a floor plan of the property and a neural network model; and atleast one processor configured to: obtain structural data of theproperty based on the floor plan; learn furnishing information byapplying the neural network model to the floor plan and the structuraldata, wherein the neural network model is trained using sample floorplans that are furnished with a variety of sample furnishing objects,and the furnishing information learned by the neural network modelcomprises one or more furnishing objects recommended for the floor plan,and the furnishing information further comprises a furnishing heat mapreflecting positions of the one or more furnishing objects to be placedin the floor plan and probabilities of the one or more furnishingobjects to be placed at the respective positions; and generate thefurnishing plan for the property based on the furnishing information, byconstructing a Monte-Carlo search tree having a plurality of tree nodeseach corresponding to a furnishing object in the furnishing heat map,and traversing the furnishing heat map subject to at least onefurnishing decision rule.
 2. The system of claim 1, wherein the neuralnetwork model is trained by a model training device, and when trainingthe neural network model using sample floor plans that are furnishedwith a variety of sample furnishing objects, the model training deviceis configured to: receive first sample floor plans, wherein the firstsample floor plans comprise at least one floor plan having a similarstructure as the received floor plan of the property; obtain samplestructural data based on the first sample floor plans; acquire secondsample floor plans corresponding to the first sample floor plans basedon the sample structural data, wherein the second sample floor plans arefurnished with sample furnishing objects; label the sample furnishingobjects in the second sample floor plans to generate sample furnishinginformation; and train the neural network model using the first samplefloor plans and the sample furnishing information generated from thecorresponding second sample floor plans.
 3. The system of claim 1,wherein to generate the furnishing plan, the at least one processor isfurther configured to: place the one or more furnishing objects tomaximize an overall placement probability of the furnishing plan,wherein the overall placement probability is determined based on theprobabilities reflected by the furnishing heat map.
 4. The system ofclaim 1, wherein the at least one furnishing decision rule is tominimize an energy function constructed based on relative positions ofthe one or more furnishing objects and framing structures in the floorplan.
 5. The system of claim 1, wherein the at least one processor isfurther configured to: generate display information that describes thefurnishing plan, the display information comprising styles anddimensions of the respective furnishing objects; and obtain accessoryobjects to complement the furnishing objects based on the displayinformation, wherein the accessory objects are selected according to atleast one of the styles or the dimensions of the respective furnishingobjects.
 6. The system of claim 5, wherein the at least one processor isfurther configured to optimize the placement of the furnishing objectsand the accessory objects in the furnishing plan based on at least onefurnishing preference.
 7. The system of claim 1, wherein the at leastone processor is further configured to: generate a 3D display model tovisualize the furnishing plan with the floor plan; and send the 3Ddisplay model to a user device for display.
 8. The system of claim 7,wherein to generate the 3D display model, the at least one processor isfurther configured to: generate a 3D property model based on the floorplan; generate 3D object models for the furnishing objects; and fit the3D object models in the 3D property model based on the furnishing plan.9. The system of claim 8, wherein the furnishing information furtheridentifies styles of the respective furnishing objects, wherein togenerate the 3D display model, the at least one processor is furtherconfigured to: determine a design style based on the styles of thefurnishing objects; and render the 3D display model consistent with thedesign style.
 10. A computer-implemented method for generating afurnishing plan of a property, comprising: receiving a floor plan of theproperty and a neural network model; obtaining structural data of theproperty based on the floor plan; learning, by at least one processor,furnishing information by applying the neural network model to the floorplan and the structural data, wherein the neural network model istrained using sample floor plans that are furnished with a variety ofsample furnishing objects, and the furnishing information learned by theneural network model comprises one or more furnishing objectsrecommended for the floor plan, and the furnishing information furthercomprises a furnishing heat map reflecting positions of the one or morefurnishing objects to be placed in the floor plan and probabilities ofthe one or more furnishing objects to be placed at the respectivepositions; and generating, by the at least one processor, the furnishingplan for the property based on the furnishing information, whereingenerating the furnishing plan further comprises constructing aMonte-Carlo search tree having a plurality of tree nodes eachcorresponding to a furnishing object in the furnishing heat map, andtraversing the furnishing heat map subject to at least one furnishingdecision rule.
 11. The computer-implemented method of claim 9, whereintraining the neural network model using sample floor plans that arefurnished with a variety of sample furnishing objects comprises:receiving first sample floor plans, wherein the first sample floor planscomprise at least one floor plan having a similar structure as thereceived floor plan of the property; obtaining sample structural databased on the first sample floor plans; acquiring second sample floorplans corresponding to the first sample floor plans based on the samplestructural data, wherein the second sample floor plans are furnishedwith sample furnishing objects; labeling the sample furnishing objectsin the second sample floor plans to generate sample furnishinginformation; and training the neural network model using the firstsample floor plans and the sample furnishing information generated fromthe corresponding second sample floor plans.
 12. Thecomputer-implemented method of claim 10, wherein generating thefurnishing plan further comprises: placing the one or more furnishingobjects to maximize an overall placement probability of the furnishingplan, wherein the overall placement probability is determined based onthe probabilities reflected by the furnishing heat map.
 13. Thecomputer-implemented method of claim 10, further comprising: generatingdisplay information that describes the furnishing plan, the displayinformation comprising styles and dimensions of the respectivefurnishing objects; obtaining accessory objects to complement thefurnishing objects based on the display information, wherein theaccessory objects are selected according to at least one of the stylesor the dimensions of the respective furnishing objects; and optimizingthe placement of the furnishing objects and the accessory objects in thefurnishing plan based on at least one furnishing preference.
 14. Thecomputer-implemented method of claim 10, further comprising: generatinga 3D property model based on the floor plan; generating 3D object modelsfor the furnishing objects; generating a 3D display model to visualizethe furnishing plan with the floor plan by fitting the 3D object modelsin the 3D property model based on the furnishing plan; and sending the3D display model to a user device for display.
 15. A non-transitorycomputer-readable medium having stored thereon computer instructions,when executed by at least one processor, perform a method for generatinga furnishing plan of a property, comprising: receiving a floor plan ofthe property and a neural network model; obtaining structural data ofthe property based on the floor plan; learning furnishing information byapplying the neural network model to the floor plan and the structuraldata, wherein the neural network model is trained using sample floorplans that are furnished with a variety of sample furnishing objects,and the furnishing information learned by the neural network modelcomprises one or more furnishing objects recommended for the floor plan,and the furnishing information further comprises a furnishing heat mapreflecting positions of the one or more furnishing objects to be placedin the floor plan and probabilities of the one or more furnishingobjects to be placed at the respective positions; and generating thefurnishing plan for the property based on the furnishing information,wherein generating the furnishing plan further comprises constructing aMonte-Carlo search tree having a plurality of tree nodes eachcorresponding to a furnishing object in the furnishing heat map, andtraversing the furnishing heat map subject to at least one furnishingdecision rule.
 16. The non-transitory computer-readable medium of claim15, wherein the method further comprises: generating a 3D property modelbased on the floor plan; generating 3D object models for the furnishingobjects; and generating a 3D display model to visualize the furnishingplan with the floor plan by fitting the 3D object models in the 3Dproperty model based on the furnishing plan.